The applicant of this application previously proposed classification adaptive processing as data conversion processing for improving the quality of images or performing other types of image conversion.
The classification adaptive processing includes classification processing and adaptive processing: data is classified by classification processing according to the property of the data, and each class of the data is subjected to adaptive processing. The adaptive processing is, for example, as follows.
In the adaptive processing, for example, low-quality or standard-quality image (hereinafter sometimes referred to as an “SD (Standard Definition) image”) data is mapped by using predetermined tap coefficients so as to be converted into high-quality image (hereinafter sometimes referred to as an “HD (High Definition) image”) data.
It is now assumed that, for example, a linear coupling model is employed as the mapping method using tap coefficients. In this case, the pixel values of pixels y forming HD image data (hereinafter sometimes referred to as “HD pixels”) are determined by using tap coefficients and a plurality of pixels forming SD image data (hereinafter sometimes referred to as “SD pixels”) extracted as predictive taps for predicting the HD pixels according to the following linear equation (linear coupling).
                    y        =                              ∑                          n              =              1                        N                    ⁢                                          ⁢                                    w              n                        ⁢                          x              n                                                          (        1        )            
In equation (1), xn indicates the pixel value of the n-th pixel of the SD image data (hereinafter sometimes referred to as the “SD pixel”) forming the predictive taps for the HD pixel y, and wn indicates the n-th tap coefficient to be multiplied with the pixel value of the n-th SD pixel. In equation (1), it is assumed that the predictive taps consist of N SD pixels x1, x2, . . . , xN.
The pixel value y of the HD pixel may be determined by equations of higher degrees, such as a quadratic equation, rather than by the linear equation expressed in (1).
When the true value of the pixel value of the k-th sample HD pixel is indicated by yk, and when the predictive value of the true value yk determined by equation (1) is indicated by yk′, the predictive error ek is expressed by the following equation.ek=yk−yk′  (2)
Since the predictive value yk′ in equation (2) is determined by equation (1), equation (1) is substituted into yk′ in equation (2), thereby obtaining the following equation.
                              e          k                =                              y            k                    -                      (                                          ∑                                  n                  =                  1                                N                            ⁢                                                          ⁢                                                w                  n                                ⁢                                  x                                      n                    ,                    k                                                                        )                                              (        3        )            
In equation (3), xn,k designates the n-th SD pixel forming the predictive taps for the k-th sample HD pixel.
The tap coefficient wn that sets the predictive error ek to be 0 in equation (3) is the optimal value for predicting the HD pixel. Generally, however, it is difficult to determine such tap coefficients wn for all the HD pixels.
Accordingly, as the standard for the optimal tap coefficient wn, the method of least squares, for example, is used. Then, the optimal tap coefficient wn can be determined by minimizing the sum E of the square errors as the statistical error expressed by the following equation.
                    E        =                              ∑                          k              =              1                        K                    ⁢                                          ⁢                      e            k            2                                              (        4        )            
In equation (4), K indicates the number of set samples of the HD pixel yk and the SD pixels x1,k, x2,k, . . . , xN,k forming the predictive taps for the HD pixel yk.
The tap coefficient wn that minimizes the sum E of the square errors in equation (4) must satisfy the condition that the value determined by partial-differentiating the sum E with the tap coefficient wn becomes 0, and thus, the following equation must be established.
                                                                                          ∂                  E                                                  ∂                                      w                    n                                                              =                                                                                          e                      1                                        ⁢                                                                  ∂                                                  e                          1                                                                                            ∂                                                  w                          n                                                                                                      +                                                            e                      2                                        ⁢                                                                  ∂                                                  e                          2                                                                                            ∂                                                  w                          n                                                                                                      +                  …                  +                                                            e                      k                                        ⁢                                                                  ∂                                                  e                          n                                                                                            ∂                                                  w                          n                                                                                                                    =                0                                                                        (                                                n                  =                  1                                ,                2                ,                …                ⁢                                                                  ,                N                            )                                                          (        5        )            
Accordingly, by partial-differentiating equation (3) with the tap coefficient wn, the following equation can be found.
                                                        ∂                              e                k                                                    ∂                              w                1                                              =                      -                          x                              1                ,                k                                                    ,                                            ∂                              e                k                                                    ∂                              w                2                                              =                      -                          x                              2                ,                k                                                    ,        …        ⁢                                  ,                            (        6        )                                                                    ∂                              e                k                                                    ∂                              w                N                                              =                      -                          x                              N                ,                k                                                    ,                  (                                    k              =              1                        ,            2            ,            …            ⁢                                                  ,            K                    )                                                
The following equation can be found from equations (5) and (6).
                                                        ∑                              k                =                1                            K                        ⁢                                                  ⁢                                          e                k                            ⁢                              x                                  1                  ,                  k                                                              =          0                ,                                            ∑                              k                =                1                            K                        ⁢                                                  ⁢                                          e                k                            ⁢                              x                                  2                  ,                  k                                                              =          0                ,                              …            ⁢                                                  ⁢                                          ∑                                  k                  =                  1                                K                            ⁢                                                          ⁢                                                e                  k                                ⁢                                  x                                      N                    ,                    k                                                                                =          0                                    (        7        )            
By substituting equation (3) into ek in equation (7), equation (7) can be expressed by the normal equations expressed by equations (8).
                                          [                                                                                (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    …                                                                      (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                                                                    (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    …                                                                      (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                                                ⋮                                                  ⋮                                                  ⋱                                                  ⋮                                                                                                  (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    …                                                                      (                                                                  ∑                                                  k                          =                          1                                                K                                            ⁢                                                                                          ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                        ]                    [                                          ⁢                                                                      w                  1                                                                                                                                                                                                                                                                                                                  w                  2                                                                                                                                                                                          ⋮                                                                                                                                                                                                                                                    ⁢                                      w                    N                                                                                ]                ⁢                                  ⁢                                                                                                                                                                  =                                                                                                                                                                                                                                                                                =                                                                                                                                                                                                                                                                                                                                                                                                        =                                                                                                                                                                                                                            ⁢                                          ⁢                                          [                                          ⁢                                                                      (                                                            ∑                                              k                        =                        1                                            K                                        ⁢                                                                                  ⁢                                                                  x                                                  1                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                                                                      (                                                            ∑                                              k                        =                        1                                            K                                        ⁢                                                                                  ⁢                                                                  x                                                  2                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                                                    ⋮                                                                                      (                                                            ∑                                              k                        =                        1                                            K                                        ⁢                                                                                  ⁢                                                                  x                                                  N                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                              ⁢                                          ]                                    (        8        )            
By preparing a certain number of sets of the HD pixels yk and the SD pixels xn,k, the same number of normal equations (8) as the number of tap coefficients wn to be determined can be found, and by solving equations (8) (the matrix at the left side next to the tap coefficients wn in equations (8) must be regular to solve equations (8)), the optimal tap coefficients wn can be determined. In solving equations (8), the sweep-out method (Gauss-Jordan elimination), for example, may be employed.
As described above, by solving equations (8) by setting many HD pixels y1, y2, . . . , yk to be supervisor data as supervisors for learning tap coefficients and by setting SD pixels x1,k, x2,k, . . . , xN,k forming the predictive taps for each HD pixel yk to be learner data as learners for learning the tap coefficients, learning is conducted for determining the optimal tap coefficients wn. By using the optimal tap coefficients wn, SD image data is mapped (converted) onto (into) HD image data by using equation (1). The above-described processing is adaptive processing.
The adaptive processing is different from mere interpolation processing in that components contained not in SD images but in HD images are reproduced. More specifically, only from equation (1), the adaptive processing is similar to the so-called “interpolation processing” using interpolation filters. However, the tap coefficients wn, which correspond to tap coefficients used in the interpolation filters, are determined by learning by using HD image data as supervisor data and SD image data as learner data. Thus, components contained in HD images can be reproduced. Accordingly, it is possible that the adaptive processing serves the function of creating images (creating the resolution).
In learning tap coefficients wn, the combinations of supervisor data y and learner data x can be changed so as to obtain tap coefficients wn performing various conversions.
If HD image data is used as the supervisor data y, and if SD image data determined by adding noise or blurring to the HD image data is used as the learner data x, tap coefficients wn for converting an image into an image without noise or blurring can be obtained. If HD image data is used as the supervisor data y, and if SD image data determined by decreasing the resolution of the HD image data is used as the learner data x, tap coefficients wn for converting an image into an image having improved resolution can be obtained. If image data is used as the supervisor data y, and if DCT (discrete cosine transform) coefficients determined by performing DCT on the image data is used as the learner data x, tap coefficients wn for converting the DCT coefficients into image data can be obtained.
As described above, in classification adaptive processing, the tap coefficient wn that minimizes the sum of the square errors in equation (4) is determined for each class, and equation (1) is calculated by using the tap coefficient wn, thereby converting an SD image into a high-quality HD image. That is, by using the tap coefficients wn and the predictive taps xn generated by the SD image, equation (1) is calculated so as to determine HD pixels forming the HD image.
Accordingly, when the dynamic range of the predictive taps is small, the HD pixels are more vulnerable to variations in the values of the predictive taps xn (pixel values of the SD pixels xn forming the predictive taps) compared to when the dynamic range is large.
For the sake of simplicity, it is now assumed, as shown in FIG. 1A, that a predictive tap consists of two SD pixels x1 and x2, and the pixel value y of the HD pixel is determined by the following equation corresponding to equation (1).y=w1x1+w2x2  (9)
Since the dynamic range of a predictive tap is determined by subtracting the minimum value from the maximum value forming the predictive tap, the dynamic range D of the predictive tap shown in FIG. 1A is expressed by the following equation.D=x2−x1  (10)
In this case, the HD pixel y in equation (9) can be expressed by the following equation from equation (10).y=(w1+w2)x1+w2D  (11)
It is now assumed, as shown in FIG. 1B, that, between the SD pixels x1 and x2 forming the predictive tap shown in FIG. 1A, the SD pixel x2 is displaced by Δx2 and is represented by x2′. Then, the HD pixel y′ can be determined from the predictive tap consisting of the SD pixels x1 and x2′ according to the following equation.y′=w1x1+w2x2′  (12)
The dynamic range D′ of the predictive tap shown in FIG. 1B is expressed by the following equation.D′=x2′−x1  (13)
In this case, the HD pixel y′ in equation (12) can be expressed by the following equation from equation (13).y′=(w1+w2)x1+w2D′  (14)
Since the SD pixel x2′ is displaced from the SD pixel x2 by Δx2, it can be expressed by the following equation.x2′=x2+Δx2  (15)
The dynamic range D′ in equation (13) can be expressed by the following equation from equation (15).
                                                                        D                ′                            =                            ⁢                                                x                  2                                +                                  Δ                  ⁢                                                                          ⁢                                      x                    2                                                  -                                  x                  1                                                                                                        =                            ⁢                              D                +                                  Δ                  ⁢                                                                          ⁢                                      x                    2                                                                                                          (        16        )            
By substituting equation (16) into equation (14), the HD pixel y′ can be determined by the following equation.y′=(w1+w2)x1+w2(D+Δx2)  (17)
Accordingly, when the SD pixel x2 forming the predictive tap shown in FIG. 1A is displaced from Δx2, the resulting HD pixel is changed from y expressed by equation (11) to y′ expressed by equation (17).
The value equal to the amount by which the HD pixel determined by a predictive tap before being displaced is changed to the HD pixel determined by a predictive tap after being displaced is referred to as the “displacement rate R”, which can be expressed by, for example, the following equation.
                    R        =                              |                                          y                ′                            -              y                        |                    y                                    (        18        )            
By substituting equations (11) and (17) into equation (18), the displacement rate R can be determined by the following equation.
                    R        =                                            w              2                        ⁢            Δ            ⁢                                                  ⁢                          x              2                                                                          (                                                      w                    1                                    +                                      w                    2                                                  )                            ⁢                              x                1                                      +                                          w                2                            ⁢              D                                                          (        19        )            
According to equation (19), the displacement rate R becomes smaller as the dynamic range D of the predictive tap is larger, and conversely, the displacement rate R becomes larger as the dynamic range D of the predictive tap is smaller.
This means that a displacement of the predictive tap hardly influences the resulting HD pixel when the dynamic range D of the predictive tap is large, and conversely, a displacement of the predictive tap considerably influences the resulting HD pixel when the dynamic range D of the predictive tap is small.
Accordingly, when the dynamic range of the predictive tap is large, the influence of a displacement of the predictive tap on the resulting HD pixel can be suppressed by the so-called “masking effect” due to the large dynamic range (hereinafter sometimes referred to as the “DR masking effect” of masking the influence of a displacement of a predictive tap on the resulting HD pixel by a large dynamic range). However, when the dynamic range of the predictive tap is small, the above-described DR masking effect does not function, and thus, a displacement of the predictive tap considerably influences the resulting HD pixel.
Thus, the quality of a resulting HD image becomes different between when a predictive tap has a large dynamic range and when a predictive tap has a small dynamic range, thereby making the user feel unnatural.
The level of the DR masking effect also changes the quality of an HD image obtained as a result of performing classification adaptive processing. Accordingly, if the level of the DR masking effect is adjustable by the user, the user is able to obtain an image having a desired quality.